Add templates for popular build backends #7857
Merged
+1,063
−53
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
samypr100
force-pushed
the
init-supports-more-build-backends
branch
2 times, most recently
from
4bc6a09 to
ed495e5
Compare
|
I tested the meson build, saw this, but I'm not sure what it means. If I build twice - the second time it will have a stale/older building twice was done something like this where the project name is I think the reason is this: the build folder has some cached stuff. And it references an include directory in |
|
Also meson uses ninja from /usr/bin/ninja in this case - is that an external dependency (not covered by build requirements?) |
@bluss I noticed the same thing and assumed this was the issue as well. I briefly searched for a way to clear the config cache via mesonpy, but didn't spent too much time on it since I'm not a meson expert. The config derived from https://github.com/scientific-python/cookie, maybe @henryiii or @rgommers have some ideas? |
|
For the ninja comment: ninja is only requested if it's not on the system (or is too old). Otherwise you wouldn't be able to support all the platforms without wheels. |
This indicates an issue: a fixed build directory (which is then reused) is pointing at a temporary build environment with a build dependency that's disappearing. That is fundamentally problematic. There are only two ways to do things correctly in principle:
Different backends may be more or less tolerant of building against build dependencies located in a tmpdir (e.g., because they redo the configure stage of the build even for incremental rebuilds), but it can't be correct to do so. I can't see from the traceback how the environment is set up and how the build backend is invoked, but I think that's where things go wrong. |
Attached debug logs for windows (in case this helps) First uv + meson build logs
Second uv + meson build logs (has errors)
meson.build
project(
'mesontest',
'cpp',
version: '0.1.0',
meson_version: '>= 1.2.3',
default_options: [
'cpp_std=c++11',
'python.install_env=venv',
],
)
py = import('python').find_installation(pure: false)
pybind11_dep = dependency('pybind11')
py.extension_module('_core',
'src/main.cpp',
subdir: 'mesontest',
install: true,
dependencies : [pybind11_dep],
)
install_subdir('src/mesontest', install_dir: py.get_install_dir() / 'mesontest', strip_directory: true) |
|
@bluss digging around meson docs for a bit, I found that |
samypr100
force-pushed
the
init-supports-more-build-backends
branch
from
ed495e5 to
a337656
Compare
rgommers
reviewed
Oct 6, 2024
I think this is not going to help beyond being a band aid for this specific scenario. For one, editable installs will be invoking For regular installs things look fine from the logs as far as I can tell. For editable installs, build isolation should be disabled for things to work well. This probably requires a bit of context, so here it is: Editable installs were not designed well for dealing with packages with compiled code. The |
All great points, thanks for the links and meson context. Maybe we could consider having a default section for meson in particular with As a starting point, I was originally of the mindset that having to use
I think there's definitely further work to improve on this area as this PR mainly is a starting cookiecutter that introduces leveraging other build backends with uv. |
samypr100
force-pushed
the
init-supports-more-build-backends
branch
from
6794774 to
33bcde2
Compare
|
If uv didn't use a different random path every time, but cached the build environment, then that would be solved, I think. Might be something to look into? It would be a big time savings for both meson-python and scikit-build-core, as they could take advantage of caching then. Meson-python should also detect a moving build environment and rerun from scratch if it's detected, a feature I recently added to scikit-build-core. |
That would be great (if possible). I've removed |
samypr100
force-pushed
the
init-supports-more-build-backends
branch
from
53753d9 to
5e4aac5
Compare
samypr100
force-pushed
the
init-supports-more-build-backends
branch
3 times, most recently
from
e811bcb to
3fbbc5d
Compare
konstin
reviewed
Oct 12, 2024
| /// Generate the `[tool.]` section of a `pyproject.toml` where applicable. | ||
| fn pyproject_build_system(package: &PackageName, build_backend: ProjectBuildBackend) -> String { | ||
| let module_name = package.as_dist_info_name(); | ||
| match build_backend { |
There was a problem hiding this comment.
Do these build backend have forever stability guarantees (somewhere in their docs/github issues where we can link them?) or do we need to put version bounds on them?
There was a problem hiding this comment.
Good question, we currently don't put bounds on hatchling either, do we want to upper bound all of them? I follow each's build backend suggested version constraints I saw on their guides.
|
Hi! Adding more build backend is a great idea and the code looks good, i left some mostly style comments. I certainly see the struggle of native build backends with the current constraints, PEP 517 left a lot missing for non-pure Python projects. From uv's perspective though, it's important that build backends work correctly as long as a valid PEP 517 environment is given (https://peps.python.org/pep-0517/#build-environment). |
samypr100
force-pushed
the
init-supports-more-build-backends
branch
2 times, most recently
from
672b931 to
a090e8f
Compare
Sounds good, should I remove |
samypr100
force-pushed
the
init-supports-more-build-backends
branch
from
a090e8f to
dda8f4d
Compare
|
(and I’ll look into meson-python soon) |
|
Then let's start with |
|
@henryiii What's the stability policy for [build-system]
requires = ["scikit-build-core", "pybind11"]
build-backend = "scikit_build_core.build" |
samypr100
force-pushed
the
init-supports-more-build-backends
branch
from
dda8f4d to
8a513ef
Compare
Done |
konstin
added
the
enhancement
konstin
changed the title
konstin
approved these changes
Oct 16, 2024
tmeijn
pushed a commit
to tmeijn/dotfiles
that referenced
this pull request
Oct 18, 2024
This MR contains the following updates: | Package | Update | Change | |---|---|---| | [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.22` -> `0.4.24` | MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot). **Proposed changes to behavior should be submitted there as MRs.** --- ### Release Notes <details> <summary>astral-sh/uv (astral-sh/uv)</summary> ### [`v0.4.24`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0424) [Compare Source](astral-sh/uv@0.4.23...0.4.24) ##### Bug fixes - Fix Python executable name in Windows free-threaded Python distributions ([#​8310](astral-sh/uv#8310)) - Redact index credentials from lockfile sources ([#​8307](astral-sh/uv#8307)) - Respect `UV_INDEX_` rather than `UV_HTTP_BASIC_` as documented ([#​8306](astral-sh/uv#8306)) - Improve sources deserialization errors ([#​8308](astral-sh/uv#8308)) ##### Documentation - Correct pytorch-to-torch reference in docs ([#​8291](astral-sh/uv#8291)) ### [`v0.4.23`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0423) [Compare Source](astral-sh/uv@0.4.22...0.4.23) This release introduces a revamped system for defining package indexes, as an alternative to the existing pip-style `--index-url` and `--extra-index-url` configuration options. You can now define named indexes in your `pyproject.toml` file using the `[[tool.uv.index]]` table: ```toml [[tool.uv.index]] name = "pytorch" url = "https://download.pytorch.org/whl/cpu" ``` Packages can be pinned to a specific index via `tool.uv.sources`, to ensure that a given package is installed from the correct index. For example, to ensure that `torch` is *always* installed from the `pytorch` index: ```toml [tool.uv.sources] torch = { index = "pytorch" } [[tool.uv.index]] name = "pytorch" url = "https://download.pytorch.org/whl/cpu" ``` Indexes can also be marked as `explicit = true` to prevent packages from being installed from that index unless explicitly pinned. For example, to ensure that `torch` is installed from the `pytorch` index, but all other packages are installed from the default index: ```toml [tool.uv.sources] torch = { index = "pytorch" } [[tool.uv.index]] name = "pytorch" url = "https://download.pytorch.org/whl/cpu" explicit = true ``` To define an additional index outside a `pyproject.toml` file, use the `--index` command-line argument (or the `UV_INDEX` environment variable); to replace the default index (PyPI), use the `--default-index` command-line argument (or `UV_DEFAULT_INDEX`). These changes are entirely backwards-compatible with the deprecated `--index-url` and `--extra-index-url` options, which continue to work as before. See the [Index](https://docs.astral.sh/uv/configuration/indexes/) documentation for more. ##### Enhancements - Add index URLs when provided via `uv add --index` or `--default-index` ([#​7746](astral-sh/uv#7746)) - Add support for named and explicit indexes ([#​7481](astral-sh/uv#7481)) - Add templates for popular build backends ([#​7857](astral-sh/uv#7857)) - Allow multiple pinned indexes in `tool.uv.sources` ([#​7769](astral-sh/uv#7769)) - Allow users to incorporate Git tags into dynamic cache keys ([#​8259](astral-sh/uv#8259)) - Pin named indexes in `uv add` ([#​7747](astral-sh/uv#7747)) - Respect named `--index` and `--default-index` values in `tool.uv.sources` ([#​7910](astral-sh/uv#7910)) - Update to latest PubGrub version ([#​8245](astral-sh/uv#8245)) - Enable environment variable authentication for named indexes ([#​7741](astral-sh/uv#7741)) - Avoid showing lower-bound warning outside of explicit lock and sync ([#​8234](astral-sh/uv#8234)) - Improve logging during lock errors ([#​8258](astral-sh/uv#8258)) - Improve styling of `requires-python` warnings ([#​8240](astral-sh/uv#8240)) - Show hint in resolution failure on `Forbidden` (`403`) or `Unauthorized` (`401`) ([#​8264](astral-sh/uv#8264)) - Update to latest `cargo-dist` version (includes new installer features) ([#​8270](astral-sh/uv#8270)) - Warn when patch version in `requires-python` is implicitly `0` ([#​7959](astral-sh/uv#7959)) - Add more context on client errors during range requests ([#​8285](astral-sh/uv#8285)) ##### Bug fixes - Avoid writing duplicate index URLs with `--emit-index-url` ([#​8226](astral-sh/uv#8226)) - Fix error leading to out-of-bound panic in `uv-pep508` ([#​8282](astral-sh/uv#8282)) - Fix managed distributions of free-threaded Python on Windows ([#​8268](astral-sh/uv#8268)) - Fix selection of free-threaded interpreters during default Python discovery ([#​8239](astral-sh/uv#8239)) - Ignore sources in build requirements for non-source trees ([#​8235](astral-sh/uv#8235)) - Invalid cache when adding lower bound to lockfile ([#​8230](astral-sh/uv#8230)) - Respect index priority when storing credentials ([#​8256](astral-sh/uv#8256)) - Respect relative paths in `uv build` sources ([#​8237](astral-sh/uv#8237)) - Narrow what the pip3.<minor> logic drops from entry points. ([#​8273](astral-sh/uv#8273)) ##### Documentation - Add some additional notes to `--index-url` docs ([#​8267](astral-sh/uv#8267)) - Add upgrade note to README ([#​7937](astral-sh/uv#7937)) - Remove note that "only a single source may be defined for each dependency" ([#​8243](astral-sh/uv#8243)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this MR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
I just hit this trying to develop scikit-learn with uv. Is this issue on your radar @konstin @zanieb ? Or should I open an issue on the tracker? |
|
@lucascolley I wasn't following this, but now I am :) A dedicated issue seems nice since we've had a couple links to this pull request already. I'm not entirely sure what needs to happen, i.e., if there needs to be a change in meson or uv. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Closes #6299
This adds basic support on
uv initto a couple additional build backends (pure and binary ones):pdm,flit,setuptools,maturin, andscikit-build-core.I didn't replace the existing
--packageflag for backwards compatibility, nor did I added a "None" build backend to imply--no-packageat this time. Instead, currently when--app --packageor--libis specified,--build-backendcan be used to change the existing hatchling backend with a different one.Since this also adds some build backend that support building binaries, I also added the generation of additional files needed for the ones supported in this PR inspired by https://github.com/scientific-python/cookie
This also allows room for the upcoming uv build backend (which I assume could become the default later on) in uv init while still allowing other ones to remain functional.
Test Plan
Added additional tests for
flit,maturin, andscikit.For backends that build binaries, only maturin will do so in tests since we have rust dev tooling already 😆.
scikitdoes not build binaries in tests as it would require additional tooling and environment configuration to be available. For these, I manually tested them on Windows and Linux.